Re-implement unread counts #7736

babolivier · 2020-06-23T18:05:07Z

Second attempt at implementing unread counts, this time without relying on push rules. ~~This is also backing out the MSC2625 implementation, which we likely won't end up using~~ reverting the MSC2625 implem has been moved to #7761.

~~Best reviewed commit by commit.~~

Fixes #7741

richvdh · 2020-06-24T09:38:09Z

This is also backing out the MSC2625 implementation

which, for linking, was #7673.

synapse/storage/data_stores/main/event_push_actions.py

Also run the linters

richvdh

shaping up, thanks.

Remember that the new table will need clearing out whenever events are purged.

synapse/storage/data_stores/main/events.py

synapse/storage/data_stores/main/schema/delta/58/09unread_messages.sql

synapse/storage/data_stores/main/events_worker.py

synapse/storage/data_stores/main/schema/delta/58/09unread_messages.sql

synapse/storage/data_stores/main/events_worker.py

richvdh

actually:

I'm generally concerned about the size of the new data, and the fact that we can never clear it out.

What if we instead added a new column to the events table which indicated if it should count towards unread count. Then calculating unread count for a given user is simply SELECT count(*) from events where room_id = ? and stream_ordering > ? and sender != ? and count_as_unread.

…er/unread_counts

stale

richvdh

looks good, but one remaining problem.

According to https://www.postgresql.org/docs/10/sql-altertable.html:

Adding a column with a DEFAULT clause or changing the type of an existing column will require the entire table and its indexes to be rewritten.

Postgres has improved this in v11, but we're still on v10 on matrix.org, and we do not want to be rewriting the events table. In any case, we need to consider other HS owners. We claim to support postgres 9.5 or later.

In short, we'll need to make the new column nullable and remove the default - and then behave sensibly when we get a NULL value back.

Unusually, sqlite is better in this area :)

babolivier · 2020-07-29T10:01:31Z

Oooh good call, I didn't think of that, thanks! Looks like both PostgreSQL and SQLite consider NULL as a false value, so simply removing the default Just Works™.

So we don't end up rewriting the whole events table when running postgres < 11.

…er/unread_counts

stale

richvdh

lgtm!

…bership_join_count * 'develop' of github.com:matrix-org/synapse: Update workers docs (#7990) Fix invite rejection when we have no forward-extremeties (#7980) Fix typo in docs/workers.md (#7992) Convert federation client to async/await. (#7975) Convert appservice to async. (#7973) Convert some of the data store to async. (#7976) Fix formatting of changelog and upgrade notes Ensure that remove_pusher is always async (#7981) Add deprecation warnings 1.18.0 Update worker docs with recent enhancements (#7969) Ensure the msg property of HttpResponseException is a string. (#7979) Remove from the event_relations table when purging historical events. (#7978) Add additional logging for SAML sessions. (#7971) Add MSC reference to changelog for #7736 Re-implement unread counts (#7736) Various improvements to the docs (#7899)

This reverts commit f23c773.

This reverts commit 8dff4a1.

The test that's being added here is the exact same one as the one introduced in #7736

* commit '3950ae51e': Ensure that remove_pusher is always async (#7981) Ensure the msg property of HttpResponseException is a string. (#7979) Remove from the event_relations table when purging historical events. (#7978) Add additional logging for SAML sessions. (#7971) Add MSC reference to changelog for #7736 Re-implement unread counts (#7736) Various improvements to the docs (#7899) Convert storage layer to async/await. (#7963) Add an option to disable purge in delete room admin API (#7964) Move some log lines from default logger to sql/transaction loggers (#7952) Use the JSON module from the std library instead of simplejson. (#7936) Fix exit code for `check_line_terminators.sh` (#7970) Option to allow server admins to join complex rooms (#7902) Fix typo in metrics docs (#7966) Add script for finding files with unix line terminators (#7965) Convert the remaining media repo code to async / await. (#7947) Convert a synapse.events to async/await. (#7949) Convert groups and visibility code to async / await. (#7951) Convert push to async/await. (#7948)

* commit '2ffd6783c': Revert #7736 (#8039)

richvdh · 2022-04-05T10:13:33Z

Note that this PR was later backed out in #8039, and re-done in #8059.

babolivier marked this pull request as draft June 23, 2020 18:05

babolivier force-pushed the babolivier/unread_counts branch from daa49fa to 956263c Compare June 23, 2020 18:10

richvdh added the feature:ftue-notifications label Jun 24, 2020

richvdh mentioned this pull request Jun 24, 2020

Support enhanced notification settings #7723

Open

babolivier removed the feature:ftue-notifications label Jun 24, 2020

t3chguy reviewed Jun 25, 2020

View reviewed changes

synapse/storage/data_stores/main/event_push_actions.py Outdated Show resolved Hide resolved

babolivier force-pushed the babolivier/unread_counts branch 2 times, most recently from 62e529d to c1b148c Compare June 26, 2020 18:37

babolivier requested a review from a team June 26, 2020 18:40

babolivier marked this pull request as ready for review June 26, 2020 18:40

babolivier force-pushed the babolivier/unread_counts branch from 7ee5086 to 4ca08f2 Compare June 26, 2020 20:00

babolivier mentioned this pull request Jun 29, 2020

MSC2654: Unread counts matrix-org/matrix-spec-proposals#2654

Open

babolivier added 12 commits June 30, 2020 16:40

Store unread messages in the database

80d03b1

Count the number of unread messages on sync requests

9db0bac

Handle redactions

e33b891

Only insert rows for local users

0ee1963

Also run the linters

Handle lack of read receipt in a room

9cb9378

Don't mark an event as unread for its own sender

9e723dd

Run linter

5399a7b

Fetch joined users from current_state_events instead of room_memberships

580b499

Send unread count to the push gateway

ff8b39d

Add a redaction helper for tests

eafdf9f

Add test case for unread counts

994f267

Run linter on tests

00fd951

babolivier force-pushed the babolivier/unread_counts branch from 4ca08f2 to 00fd951 Compare June 30, 2020 15:41

babolivier added 2 commits July 2, 2020 15:36

Implement the latest changes in the MSC

a008f6f

Merge branch 'develop' into babolivier/unread_counts

fe5352f

richvdh suggested changes Jul 3, 2020

View reviewed changes

richvdh previously requested changes Jul 3, 2020

View reviewed changes

babolivier added 2 commits July 27, 2020 16:48

Lint

6f50007

Merge branch 'develop' of github.com:matrix-org/synapse into babolivi…

b265bef

…er/unread_counts

babolivier requested a review from a team July 27, 2020 17:48

richvdh previously requested changes Jul 28, 2020

View reviewed changes

babolivier added 2 commits July 29, 2020 11:02

Remove default value for count_as_unread

debedb3

So we don't end up rewriting the whole events table when running postgres < 11.

Merge branch 'develop' of github.com:matrix-org/synapse into babolivi…

57ded3e

…er/unread_counts

babolivier requested a review from a team July 29, 2020 10:04

richvdh approved these changes Jul 29, 2020

View reviewed changes

babolivier added 3 commits July 29, 2020 16:30

Match the latest changes on the MSC

82d9f39

Fix test

17e922a

Typo

d7fdc35

babolivier merged commit 8dff4a1 into develop Jul 29, 2020

babolivier deleted the babolivier/unread_counts branch July 29, 2020 17:26

babolivier added a commit that referenced this pull request Jul 29, 2020

Add MSC reference to changelog for #7736

f23c773

babolivier added a commit that referenced this pull request Aug 6, 2020

Revert "Add MSC reference to changelog for #7736"

fc721a5

This reverts commit f23c773.

babolivier added a commit that referenced this pull request Aug 6, 2020

Revert "Re-implement unread counts (#7736)"

1471fe1

This reverts commit 8dff4a1.

babolivier mentioned this pull request Aug 6, 2020

Revert #7736 #8039

Merged

babolivier added a commit that referenced this pull request Aug 6, 2020

Revert #7736 (#8039)

2ffd678

babolivier mentioned this pull request Aug 11, 2020

Re-implement unread counts (again) #8059

Merged

babolivier added a commit that referenced this pull request Aug 11, 2020

Add some tests and fix the existing ones

c8f38f6

The test that's being added here is the exact same one as the one introduced in #7736

babolivier pushed a commit that referenced this pull request Sep 1, 2021

Merge commit '2ffd6783c' into anoa/dinsic_release_1_21_x

d47db9e

* commit '2ffd6783c': Revert #7736 (#8039)

richvdh mentioned this pull request Apr 5, 2022

Implement unread counter (MSC2625) #7673

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Re-implement unread counts #7736

Re-implement unread counts #7736

babolivier commented Jun 23, 2020 •

edited by richvdh

Loading

richvdh commented Jun 24, 2020

richvdh left a comment

richvdh left a comment

richvdh left a comment

babolivier commented Jul 29, 2020

richvdh left a comment

richvdh commented Apr 5, 2022

Re-implement unread counts #7736

Re-implement unread counts #7736

Conversation

babolivier commented Jun 23, 2020 • edited by richvdh Loading

richvdh commented Jun 24, 2020

richvdh left a comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

babolivier commented Jul 29, 2020

richvdh left a comment

Choose a reason for hiding this comment

richvdh commented Apr 5, 2022

babolivier commented Jun 23, 2020 •

edited by richvdh

Loading